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ABSTRACT 

A review of the 'grading practices of various 
departments in the Western Australian Institute of Technology is the 
topic of this paper.* The., study was initiated in 1969, when an 
examination of scores given by various departments revealed a large 
year-to-year fluctuation. It was noted that some departments 
consis'tently graded higher than others. A historical account of the 
study, with graphs .showing the grading inequities, is provided; Tiie 
inception of a "standard grading distribution" and problems *hich 
resulted from its implementation are also described. Due io the 
number of arbitration 'situations in which students questioned Jbheir 
grades^ a computer program was developed which made examination and 
student academic information available to arbitrators. The author 
states that his addition of an easily accessible information network 
hag greatly increased the efficiency of the grading system^ (CP) 
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The Use of Students* Academic Histories and the Computer t6 Improve 
the Accuracy of Assessing Academic Performance of Students. 



SUMMARY : 
Several years ago investigations at the Western Australian 
Institute of Technology showed that scores in certain subjects with 
large student numbers varied widely from year to year. It was 
further noted that, in any one year, scores obtained in certain 
subjects varied widely from those in allied subjects where similar 
performance could be expected. ^^V^ 

This caused concern. Were the variations due to faulty teaching, 
faulty assessment techniques, a. faulty cijrriculum, or even faulty 
students. 

As a result the Institute reviewed i'ts scoring processes and 
brought down guidelines f'pr score distributions. These were 
intended to Identify problem, subjects so that the causes of the 
troubles could be determined and appropriate remedial action taken. 

f 

Some lecture^•s accepted the guidelines as they were Intended and 



assessrjent* processes improved. Others constde^ed the guidelines 
an infrJngement of "academic freedom" and reacted accordingly. Yet 
again a third group adjusted scores to conform to the distribution 

c 

without giving real thought to the educational reasons for doing 
this. / - 

i ' ' ' 

One of the problems in utilising the considerable volume of avail- 
able'material to improve assessment vyas a time limitation between 
examinations and the issuing of results. Subsequently a computer 
program was prepared so that the information required could be 
provided in the available time span. 

This has had considerable impact. Many of the opponents of the 
guideline scheme have now changed their attitude and' a marked 
improvement of the assessment procedure has resulted. 

Details of th^ development of -the system, the computer printouts 
made available, and the benefits observed are considered in the 
paper. 

1 INTRODUCTiON 

The author's experience with student assessment at the tertiary 
level *in the .USA has been restricted to the post-graduate level. 
A point that impressed was the expediency with which results were 
processed (a post card indicating the grade achieved usually arrived' 
within 48"hours) and the fact that grades were generally higher than 



those given in Australia. 

At the undergraduate level in Australia, and in particular at the 

Western Australian Institute of. Technplogy, a detailed analysis of 

s tudenrtSN^performance is undertaken. Before any results are issued 

the performance of students in all subjects is recorded and a Board 

H * i 

V 

of Examiners reviews overall performance. If the overall performance 
of a student is good but that in a particular subject is below the 
pass- level he may be granted a conceded pass or a^Tfeast a supp- 
lementary examination in that subject. , 

.The advantage of the system Is that it looks at the overall 
performance of a student in his fieVd. Unfortunately the procedure 
also has short-comings. In a 1969^ study it was observed that the 
year by year score variation in some subjects with large student 
populations was much greater than expected. These changes could not 
be accepted, particularly when the score variations in allied 
subjects with similar students and numbers were comparatively small. 
Figure 1 i 1 1 ustrates the score range obtained in one such subject 
.and in another subject. - ♦ 



^ Ref. 1, p. 38^ 
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FIQu\e 1-VARiATION OF AVERAGE SCORES AND PERCENTAGE 
OF STUpENTS PASSING IN PARTICULAR /lyNNUAL EXAMINATIOflS 

6 . 



' What was the cause? Taulty curriculum, faulty assessment Vocedur^,? 
faxjlty teaching, student /nal performance or something else? 

Institute senior management decided that steps should be taken 
to eliminate or at least minimi^se the problem. 



^2 THE PROBLEMS 

' It Is reasonable to expect that in large classes, the range of 

abi 1 ity, should be reasonably constant from yeSr to year. Thus 

assessment scores should also be reasonably constant if examinations 

2 

measure the students* ability accurately and teaching, is consistent 
from year to year. 



Likewise it was expected that examination score distribution 

Institute wide would be reasonably constant from year to year. This 

3 

could provide a distribution pattern for individual subjects. 



The Institute distribution of scores was determined for 19^7 
and Is as in figure 2. 



2 ^ ■ k) ' 

With small classes the range of student abi 1 i ty could vary 

considerably from year to year giving rise to varying examination 
scores. 

The student population of the Ijjstitute is- currently of the 
order of 10,000. 



After wide-ranging discussions the Institute's management group 

« 

accepted^the following frequen^zy distribution guidelines for 
examiners. These were based on the information in figure 2 and 
other. material aval lable,. 



TABLE 1 

GUIDELINE DISTRIBUTION OF SCORES IN INSTITUTE EXAMINATIONS 

Examination Score Grade Percentage Population 

Range in Range 



75 - 


100 


A 


0-25 


65.- 




B 


10 - ko 

1 


50 


ek 


C 


30 - 65 


liO - 






0-20 


20 


39 


G ^ 


0-10 


0 - 


19 


R 


excluded 



\ 



These figures reflect a desire to improve the performance of 
that s6fetion of the "tail" which performed reasonably well b^t to 
exclude the group who scored in the 0-19 range as it was (considered 
these were not really sincere students. Thus the percentages^! n 
column 3 bf Table 1 were to be applied to the sample remaining after 
the "R" group had been excluded. 



Students who scored less than 50 failed the examination. 



3 IMPLEMENTATION 

The proposals wej^implemented in 1970'. Where examiners had 
reasonable backgrounjJ in educational theory, thus* appreciating the 
limitations associated With assessment procedures and having know- 
ledge of scaling techniques, there was widespread acceptance of 
the scheme. * 

However, a second group asserted their right to ''protect the 
profession'* and fail students when their raw score was below 
50 per cent. This could result in a student who scored^S^ being 
failed even though the highest score achieved in the examination 
may have been 60^ and where scores of the same students in similar 
subjects ranged to maximums, in the nineties. 

A^third group took the^easy Way out by aidjusting scores to 
con form to' the guidelines withogt considering why they were doing 
so. They were simply conforming with management policy. ; . 

Thus the first group of lecturers and their students benefited 
from the program but the same could not be s^id of the others. 

The second group in particular caused considerable concern and 
it was decided that if the examiner and head of department could 
not agree with the Dean regarding the need to adjust particular 
scores the matter would be- referred to an arbitration group. 



10 



o 



'a 



The proceedings of the arbitration group indicated that, to some 
examiners, the scoring process was precise and objective. It -was of 
concern to realise that the inherent limitations and problems were 
not appreciated. 

*. 

' A typical case will illustrate the procedure at an arbitration 
hearing. In a particular subject an unduly larAe percentage of 
students had failed. The examiner, supported by prf« head of depart- 
ment, argues that he is justified in retaining this pass rate because 
the students had failed in other subjects. The following facts are 
brought forward by the Dean to justify a revision of scores. 



i. The particular student group had similaV course entry scores 
to engineering students in other departments. TheseN^her 
students had performed well, 

n. The performance of.fhe student group in common subjects such 
as mathemati*cs and physics was similar to that of other 
engineering stucfen^s. 

*■ • 

i?i. The scores' for the grdup under consideration were depressed 

in comparison with those In previous years. 

/ ^ 'i 

iv. Investigation of performance on individual questions In the 

At 

examination showed there had been two" questions with 
universally poor scores\ 
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The department hdd ^t obtained th^ls informatlorv and hence it ^ 
could not make a true evaluation. ^ In fact the student definitely 
was not at fault I 



Un/ortunately an appeal court Is not the place for major adjust- 
ments to be made. Prejudices prevail and it is difficult to effect 

» 

changes. Consequently th^gh some improvements were made, much was 
left^to be desired."* 

/ 

k, UPGRADING THE SYSTEM 

The policy as detailed continued through 1971 and 1972 with fairly 
regular confrontation betvyeen Deans, Heads of Departments and 
Examiners. It became apparent that much more could be'achieved if 
the Boards of Examiners could have before them the detailed Analysis 
of performance and historical records that were labbf-icjwsly produced 
manually for the confrontation sessions. 

The computer was the logical machine to expedite the provision 
of this Information. It was already used to produce ^printout 
.showing the performance of each student In all subjects sat In an 
examination. ^, 

* 

Mr. S. Nowak, a member of the Mechanical Engineering Department, 
was commissioned to investigate the matter and prepare appropriate 
computer programs* 
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Experience hBd shown»that the following deficiencies were 
associated with the existing assessing scheme: 

-•V ^ , • 

I. partl'cular examiners considered student scores in' their. 
♦ 

Subject in i sola t Ion. 

o 

ii. The performance of individual^ students as well as that of 
* the group in a particular subject were not compared jyl th 
performanc^es In related subjects. 

ill. Generally a. student's performance in a particular subject 

vms not compared with the overall performance of students tn 
,;his class (or his group whire there was more jthan one teaching 
group) . ' 

iy. No analysis was conducted to alscertaln the relative performance 
of all students on individual questions in an examlnat^idn 
pap'er. 

v. No attempt was made to determine [f there Kad beett.'C<SiS;Istency 
of syllabus treatment wh^e more than one lecturer handled 
the subject. " . . ' > 

vT. . The. earJrier performance of a student and his course entrance 

^ scores were not readily available to Boards ^of Ex;amihers ^ 

' . * *• • 

•when assessing the student's performance. 



5 THE UPGRADED SYSTEM 

Yhe computerised system analysing students' performance was 



subdfvlded into thVee -inter-related parts: 

I. The preparation by the computer of mark sheets. 

* 

ii. The 'analysis of examination results from th^ irjformation 

♦ * 
submitted on mark sheets. 

lii. Printouts providing information on which to Judge the students 
present and future status at the Institute. 

The three separate computer programs prepared for these steps are 
now considered in detail. 

5. 1 Mark Sheets 

The computer prints out separate sheets for each su'fij^ct being 
examined. These list the enrolled students alphabetically and also 
provide cpl.umns for scores obtained In each question In the end of 
semester iexaminati on, semester scores for laboratory work, assign- 
ments and tests, and information as to whether the student is 
attempting the subject for the first, second or thjrd time. The 
sheets are custom made to fit the requirements of particular depart- 
ments, a typical sample partially completed. Is shown in figure 3. 
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The column GRP Is used to indicate the group number. Thus if the 
-c+a« fs divided into four teaching groups these would be rderrttf ied 
at GRP 1, 2, 3 and 4. • ' 

Question scores are entered in columns Q1 , Q2 and so on, whilst 
the ASG and LAB columns take assignment and laboratory scores. The 
final column, COM is used to record the number of attempts a student 
has had in a particuJar subject. 



It has been found useful to know how many students actually 
attempted a particular question as this had obvious implications 
regarding Its difficulty or the effectiveness of the treatment of 
that particular part of the syllabus. Consequently examiners were 
instructed to enter a mark of 0.1 where a student had attempted a 
question but obtained zero marks, and jeave a blank if the question 
was not~attempted . 

4 / 

n 

Provided teach ing"'dj|fi^Vtments submit the information in time 
mark sheets can be produced at the beginning of a semester. They 
are then particularly valuable where continuous assessment Is belng^ 
used. Alternately the sheets can be provided at a later. date as 
required. Irrespective of when the sheets are. provided the relevant 
material must be entered and returned to the computer center for 
processing immediately after the final semester examinations. 

16 

t ' — ..^ 




5.2 Analysis of Examination Results . «^ 

Prior to the examination the computer has be^n advised bf the, 
number ,of questions In the particular examination paper,^ the number 

of questions to be attempted, the maximum fnarks for ^ach question, 

■» * 
and for assignments, laboratory work and continuous assessment 

programs. The weighting to be giVen to the end of se'mester *• ' 

examination and the other compon^iiis of the assessment t)rocedure ^ 

are also programmed in. Thus when the computer repeives the 

information from subject mark sheets, via punched cards, it can 

immediately print out the raw score and correspond! ng *grade (O 9) 

4 . 

for each student as In figure 4. Additional information is also \ 
provided then as shown in figure 5> which'detaMs the followihg. 



I. '^the average score and the standard deviatfon for the subject.- 

ii. The distribution of grades for students new to the subject 
and for those repeating for the first and second time. ^ 



ii. The 'performance of the class as a whole on various ^ 
, individual questions. 



An addi(tiortal similar printout givjss the information Of Items 
I and ii above^for the. various teaching groups. within the class* 
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Figure 6 shows an alphabetical list of all students in the 
subject together with their final raw score and corresponding grade. 
Provision is made for amendments or recommendation by the Board of 
Examiners. The final print out prepared from the amended figure 5 
gives an alphabetical list containing the final approved grade. 



Some comment regarding the use of this information is appropriate. 



Figure 3 i ndi cates' that question one was unpopular as it was 
avoided by an unexpectedly high portion of the population. It also 
shows that performance on, this question was generally poor. The 
examiners are thus alerted to investigate it. Is this topic 
traditionally unpopular - and if so what can be done about it? 
Or were the results due to a poorly set or ambiguous question? 
Yet again were they due to poor preparation of all students In the 
subject or of those in one or more groups* wi thl n the class? . 

Having identified the cause appropriate remedial action can be*, 
taken^for the present and the future. For example if the difficulty 
lis due to a poor?y set examination question some scaling of scores 
is indic'atexL )lf the difficulty is related to the teaching of the 
group as a\hoJle some form. of scaling along with a review of the 
relevant curriculum section and teaching method should be undertaken. 
Where only some groups have experienced difficulty, there. is a case 
for differential scaling together wtith a review of the teaching 
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methods of lecturers involved. Thus it is possible to compensate* 
for short term problems in examination scoring which would otherwise 
penalise the student unfairly. At the same tincie remedial action 
can be taken to avoid or m^inimtse the problem in the future. ' 

Questions 2 and 3 were badly answered. The examiners must 

determine if the questions were ambiguous or if students lacked 

preparation. Or can some other reason for the poor results be 
identi f ied? ^ 

The distribution of grajdes in Figure 5 shows that 57,^ percent of 
sfudents scores are. between 50 and 70 percent and the average score 
of 58? for the subject appears to be reasonable. .The scatter at 
15-8 percenl&gmay be a little high. Why is the scatter high? 
Referring to figure h it is noted there appears to be a difference 
between the resul ts. obtaf'rtecl in the examination and the overall result, 
The weighted marks for assignments (maximum score of 30) indicate all 
students performed well on assignments but poorly In examinations 
(maximum score of 70). Was the weighting for examinations too high - 

0 

or the marking of assignments too generous? If the assignment^ is a 
reasonable reflector of the students ability, soinething is obviously 

s 

wrong with the examination. 

In addition to the advantages discussed the system aids in achieving 
uniformity of standards where there are several different groups in 
the subject and also enables a feedl^ack report to be prepared showTng 
thQ performance of a student in relation to that of others in the 
subject. Finally the system permits of the validation of the test 
Instrument Msed, and where it is a reusable test, to suggest areas 
where changes are required. 22 
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Determination of Final Grades and Student Status 



In doing this, the following information is provided to examiners 
and boards of examiners* 



/ 



L The performance of the stuBeift as an individual (fi-gur^ 4). 

n. The performance of the student within a' group or class in 
a particular subject (figures k and 5). 

iii. The performance of the student in all subjects in the 
semester (figure 7). 

Additionally, earlier history records of 1:he student are printed 

4 

out (figure 7) . 

r > 

To indicate the relative standing of various students in the 
subject by .comparison with the year reference mark, raw scores are^ 

■a '''' ; 

scaled so that the subject average and the average for all students 
in. all subjects in a particular semester of ,study are the same. 
These scores and grades are shown in columns 3 and 4, figure 7. * 
In. this program sc'allng does not equalise fhe standard deviations 
for the subject and for the year. Thus,*there could be criti^oism 
that the scaling process does not go far enough. However, it does 
serve as a guide as is intended* 



,lt should be stressed that this automatic scaling is done so 

23 
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that it is possible to assess overall performance meaningfully and to 
help in the development of 'the students future program. The Board of 
Examiners or the individual examiners are under no compulsion to 
use the scaled scores. They can elect to leave. the score unchanged 
if deliberation suggests this is the wisest course. They can use 
the scaled score, or they can adopt another score as determined by • 
the circumstances. 

Figure indicates the type of information provided to assist 
examiners in deciding on final scores. Consider case 13 in figure 7. 
The first column gives the raw score obtained by the student in each 
exam in the semester together with his average. The final column 
gives the semester average for all subjects and students in the 
identified semester subject group. The third column gives the scaled 
score and colum A the scaled grade. Thus for subject E.C. I90 

average score for prescribjed 
Scaled score = subject raw score x subjects i n semester of study 

average for the subject 

Finally the leaving and matriculation scores (alternate course 
entry examinations) are given. 
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to all Institute departments. Thus the first evaluation was 
restricted to the four departments of the School of Engineering 
and Surveying. ^ • * , 



There has been a marked decline, to niV in 197^, in the number? of 
disputed cases referred to arbitration. Again, departments which 
had opposed scaling in any form now accept that some form of scaling 
IS required and this is done as a matter of course'. • • 

Originally the examining process was largely one of transferring 

raw examination scores in the shortest possible time from the 

department to the examinations office. Now, the best is made of 

5 

the limited time available after the examinations and Judgements 
are made regarding the students' performance using all possible 
supporting information which is now madfe available. 

Late in 197^ a report of the investigation, prepared by the ^. ^» 
writer and Mr. NowakT^s presented to the Institute's Academic 
Board which determines educational policy. This Board, in evaluating 
the report, agreed that 

^ In 197^ second semister examinations, the examination perfod 



extended over seven 
before al 1 material 



jays followed by a period of nine working days 
lad to be submitted by examiners. All computer 
printouts for Boards of Examiners were available three actual days 
fater* 
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"Academic Board commends the report to the working^ party on 
degrees with distinction^ and reconmends the procedures 
contained therein to interested departments." 

It !.s- hoped the procedure will be widely implemented.' . 

the -fol lowing points need further revlewlr\g as the scheme is 
developed. • * . 



Mention hds be^n made of the fact that the scaling process 
does not correct for variations in score, spread. There 
could be vslue in ascertal ning the effect jQf-^s second 
'form of seal ing. 

There are cases on record where scores in a majority of 
subjects 'in a particular year of a course have been depressed 
, and investigation ha^ s.yggested that this was due to reasons 
other than student ability. In the present scaling system 
used this could lead to erroneous results as" ^fiese depressed 
semester scores are used to obtain the scaling average. It • 
is suggested that more meaningful results cpuld be obtained 
were the subject averages to be determined considering scores 
overa number of years. ' * • 
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Ill, Attempts are being m^de to reduce the time of processing 

-^results. by making use of a' document reader* It Is proposed 
that a pilot scheme should be Introduced., in 1975. The 
programme Fn its present form uses data from punched cards. 
As an alternate data can be read directly using the document 
reader. A major advantage here would be the elimination of 
the intermediate punching process in the transfer of 
Information. This will reduce possible errors, expedite the 
process, and reduce the skilled labour component of the^process 
" " thus effecting economies. 
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